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ABSTRACT  improving  strategic  mission 

implementation. 

This  paper  offers  an  approach  for 

performance-based  decision  making  and  INTRODUCTION 
budgeting  that  objectively  orders 

alternatives  according  to  established  Decision-making  that  optimally 

mission  criteria.  We  employ  a  model  allocates  scarce  resources  has  always 

using  Value-Focused  Thinking  (VFT)  to  been  a  critical  management  skill, 

address  the  contemporary  focus  on  Recently,  selecting  alternatives  that  have 

performance  and  mission-oriented  the  most  impact  on  the  organization’s 

results.  The  laboratory  equipment  mission  has  also  become  a  paramount 

budget  at  the  US  Air  Force  Academy  is  concern.  Unfortunately,  developing 

used  to  illustrate  how  over  $1  million  of  objective  measures  that  prioritize 

funding  was  allocated  without  the  alternatives  when  mission  statements  are 

divisive  claims  of  departmental  inequity.  subjective  and  vague  is  problematic. 

We  provide  an  objective  rubric  for  However,  a  renewed  focus  on  mission- 

assessing  how  well  aligned  the  oriented  results,  ushered  in  by  a  host  of 

organization’s  decisions  are  relative  to  federal  legislation  and  popular 

its  stated  mission.  Finally,  we  note  that  management  philosophies,  has 

the  approach  accommodates  the  compounded  the  significance  of  the 

flexibility  to  allocate  initial  budget  challenge  (Figura,  1 999).  In  many  cases, 

allocation  as  well  as  year-end  residual.  time-proven  analytical  approaches  do  not 

As  organizations  make  significant  strides  provide  adequate  information  for 

toward  implementing  performance-based  decision-makers.  More  than  ever  before, 

decision  and  budgeting  systems,  VFT  decision-makers  must  temper  their 

shows  considerable  promise  for 
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decisions  based  upon  both  internal 
organizational  consequences  while 
meeting  the  demands  of  external 
stakeholders.  This  emerging  focus  on 
outputs,  outcomes,  and  results,  in  the 
context  of  strategic  mission 
implementation,  is  the  focus  of  this 
paper. 

We  offer  an  approach  that 
satisfies  the  needs  of  internal  and 
external  stakeholders.  The  approach 
provides  a  sound  analytical  foundation 
for  complex  strategic  decisions,  while 
offering  an  objective  rubric  for  assessing 
how  well  aligned  the  decision  are  to  the 
organization’s  mission.  The  approach 
may  be  used  for  a  broad  range  of 
situations  where  allocation  of  scarce 
budgetary  resources  is  complicated  by 
competing  projects,  all  of  which 
contribute  to  organizational 

performance.  This  type  of  performance- 
based  budgeting  requirement  will 
undoubtedly  drive  the  decision  making 
landscape  for  the  foreseeable  future. 

Strategic  decision  tools  that  rely 
upon  and  assist  the  development  of 
performance  metrics  are  especially 
valuable  within  the  DoD  and  other 
government  agencies.  Legislation 
compliance  is  compounded  by  the 
explicit  need  to  improve  taxpayer 
confidence  by  holding  agencies 
accountable  for  achieving  results.  The 
1993  Government  Performance  and 
Results  Act’s  (GPRA)  primary  purpose 
is  to  improve  decision  making  and  the 
internal  management  of  the  government. 
It  shifts  the  focus  of  federal  agencies 
away  from  traditional  concerns  of 
staffing  and  activity  levels  toward  a 
single  overriding  issue:  results  (GAO, 
1996).  While  performance  promises 
have  accompanied  funding  requests  in 
the  past,  GPRA  explicitly  mandates  the 


financial  consequences  of  not  fulfilling 
stated  promises.  Organizations  must 
reflect  specific  results  that  are  visible 
and  reportable  to  legislators  and 
ultimately  taxpayers  (Comes  and  Riley, 
1999).  If  these  results  are  not  aligned 
with  the  organization’s  mission,  funding 
might  be  adversely  affected,  and  the 
value  the  organization  provides  might 
come  into  question.  In  fact,  according  to 
the  Office  of  Management  and  Budget 
(OMB),  which  is  responsible  for 
implementing  and  monitoring  the 
GPRA,  “...eventually,  the  annual 
performance  plan  will  be  integrated  with 
the  agency’s  budget  report.”  (OMB, 
1998).  Goals  and  outcomes 

accompanying  organizational  mission 
statements  must  be  measurable,  and 
eventually  linked  to  specific  budget 
requests.  Observers  contend  that 
legislators  barely  disguise  their 
intentions  to  use  this  type  of  information 
as  ammunition  to  reduce  federal 
manning  levels  (Laurent,  1997). 

To  complicate  matters,  managers 
must  still  make  decisions  that  are 
sensitive  to  intra-organizational 
dynamics.  They  must  make  resource 
allocations  that  might  spark  criticisms  of 
shortsightedness  or  favoritism.  If 
performance  of  activities  cannot  be 
measured  or  aligned  with  organization 
missions  or  goals,  then  the  value  of  that 
activity  should  come  under  scrutiny. 
Ideally,  decision-makers  can  approach 
situations  with  a  tool  that  addresses  both 
internal  and  external  stakeholders. 

Our  approach  uses  Value 
Focused  Thinking  to  provide  the  credible 
information  required  to  immediately  put 
into  action  a  performance-based  decision 
and  budgeting  process  that  considers 
both  internal  and  external  perspectives. 
It  allows  an  organization  to  take  that  first 
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step  toward  implementing  a  mission- 
oriented  budgeting  and  management 
system. 

VALUE-FOCUSED  THINKING 

Multi-Attribute  Decision  Analysis 
is  specifically  designed  to  meet  the  need 
for  an  objective  means  assessing 
resource  allocation  scenarios  (Keeney, 
1976,  Hwang  and  Yoon,  1981). 
However,  while  the  methodology  helps 
in  efficiently  allocating  resources 
between  competing  objectives,  it  does 
not  guarantee  the  “right”  objectives  were 
identified  and  defined.  Value  Focused 
Thinking  (VFT)  focuses  attention  upon 
organizational  values,  which  can  be 
translated  into  objectives.  This  approach 
places  value  on  mission  effectiveness 
and  objective  measures.  It  prescribes 
importance,  which  allows  decision¬ 
makers  to  shape  the  future,  and  appears 
to  be  ideally  suited  for  long-range, 
guidance-type  decisions  (Keeney,  1992). 
VFT  requires  the  judgement  of  experts 
regarding  future  values  and 
interpretation  of  organizational  missions 
and  goals.  Because  they  are  mission,  as 
opposed  to  profit  oriented,  this 
methodology  seems  ideally  suited  for  the 
public  sector. 

The  United  States  Air  Force 
recently  employed  VFT  to  evaluate  the 
value  of  future  air  and  space  forces  in  the 
AIR  FORCE  2025  Study.  The 
Foundations  2025  model 

“...successfully  scored  43  futuristic 
system  concepts  ...  on  134  attributes  ... 
and  provided  insights  about  the  most 
promising  system  concepts  and 
technologies.”  (Parnell,  et  al.,  1998). 
This  technique  was  also  successfully 
employed  for  Spacecast  2020,  a  study  to 
identify  future  space  system  concepts 


(Burk  and  Parnell,  1997).  Another 
recent  study  involving  VFT  evaluated 
the  past  performance  and  prescribed 
future  efforts  of  AFROTC  detachments 
(AFROTC  Viability  Study;  Hague, 
Newton,  and  Lowe,  1998).  All  of  these 
large-scale  successful  applications  of 
VFT  add  to  the  track  record  of  providing 
a  sound  analytic  foundation  for  complex, 
strategic  decision-making. 

A  multi-attribute  approach  was 
employed  to  maximize  benefits  in  the 
repair  of  Canadian  Forces  land-based 
equipment  (Mitchell  and  Bingham, 
1986).  Likewise,  a  hierarchical 
approach  was  used  to  generate  objective 
function  weights  necessary  to  support  a 
large-scale  goal  program  model  of  a 
multi-year  Army  Personnel  Planning 
Model  (Gass,  1986). 

The  original  hierarchical  model 
described  in  this  paper  was  motivated  by 
the  Analytic  Hierarchy  Process  (AHP), 
developed  and  applied  by  Saaty  (1980, 
1982,  1990).  “The  AHP  is  useful  in 
priority  setting  (ranking),  but  it  is  its 
extension  to  planning  and  resource 
allocation  decisions  where  it  may 
ultimately  have  its  greatest  utility” 
(Liberatore,  1987[1],  p.13).  However, 
for  the  case  study  described  in  this  paper, 
the  decision-makers  were  comfortable 
using  a  modified  Delphi  Process,  to 
directly  assess  the  hierarchy.  Within  the 
Delphi  Process,  they  relied  upon  pair¬ 
wise  comparisons  to  determine  several 
attribute  measures. 

Liberatore  relied  upon  the 
AHP/Optimization  approach  in  creating 
a  decision  support  system  to  allocate 
funds  to  R&D  projects  (Liberatore, 
1987[2]).  While  his  hierarchical 
development  and  attainment  of  attribute 
weights  were  similar  to  our  approach,  a 
difference  was  that  he  included  financial 
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considerations  (return  on  investment, 
market  share,  etc.)  within  his  value 
hierarchy  and  weighted  these  aspects 
along  with  other  desirable  R&D 
attributes.  Our  model  kept  cost  and 
budget  considerations  out  of  the  Value 
Hierarchy,  but  included  these  concerns 
as  posteriori  constraints.  The  potential 
impact  of  this  modeling  difference 
cannot  be  predetermined  (Bernhard, 
1990).  Both  approaches  eventually 
assign  a  “weight  to  cost.”  However,  our 
model  relies  upon  an  optimization  to 
create  the  “opportunity  costs”  of 
constraints  (budget)  rather  than  asking 
decision-makers  to  make  the  tradeoff  a 
priori.  French  (1988,  1989)  provided  the 
theoretical  basis  which  guided  our 
modeling  strategy  (see  also  Melese, 
Lowe,  and  Stroup,  1997). 

Finally,  sensitivity  analysis 
provides  comfort  to  stakeholders  by 
illustrating  the  effect  upon  the  final  unit 
rank  order  of  changes  to  the  attribute 
weights.  Vazsonyi  (1995)  and  Kirkwood 
(1997)  present  practical  examples  of 
performing  and  presenting  sensitivity 
analysis  results.  Vazsonyi’s 

“performance”  profile  ideas  formed  the 
basis  of  the  sensitivity  charts  used  by  our 
approach. 

CASE  STUDY:  THE  DEAN’S 
EQUIPMENT  BOARD  (DFEB) 

The  Dean  of  the  Faculty  at  the 
United  States  Air  Force  Academy  is 
faced  with  many  diverse  resource 
allocation  decisions.  One  such  decision 
involves  the  disbursement  of 
approximately  $1M  for  laboratory  and 
other  equipment.  Accordingly,  this 
disbursement  requires  an  “equitable” 
distribution  of  budget  authority  among 
widely  disparate  requests.  Historically, 


the  process  has  been  emotionally 
charged  due  to  the  reported  urgent  nature 
of  almost  all  of  the  requests  and  the 
seemingly  constant  reduction  of 
available  funds.  To  provide  a  non- 
parochial  approach,  an  Equipment  Board 
(the  DFEB)  was  established  with 
representatives  from  each  academic 
division  to  equitably  allocate  funds 
across  the  faculty.  This  allocation  was 
not  straightforward;  equipment  requests 
came  from  19  departments  and 
supporting  staff  agencies  with  widely 
varying  requirements. 

By  the  nature  of  their  academic 
disciplines,  the  9  departments  in  the 
Social  Science  and  Humanities  Divisions 
were  less  inclined  to  have  requests  that 
approached  the  dollar  figures  of  the  10 
departments  found  in  the  Engineering 
and  Basic  Science  Divisions.  However, 
organizational  climate  could  be 
adversely  affected  if  requests  were 
determined  by  relative  dollar  size  alone. 
What  was  required  was  an  objective 
means  of  prioritizing  the  equipment 
budget  allocation  that  was  efficient, 
credible,  traceable,  relevant,  and 
considered  fair  by  the  competing 
department  heads.  Originally,  the  DFEB 
considered  several  factors  in  allocating 
funds.  To  its  credit,  the  DFEB  rejected 
the  straightforward  but  ill-considered 
method  of  simply  dividing  the  funds 
equally  across  departments.  The 
limitations  of  this  approach  are  well 
documented  ( e.g .,  Melese  and  Lowe, 
1992). 

The  DFEB  ultimately  decided  to 
use  a  VFT  approach,  which  placed  value 
on  objective  measures  contributing  to 
mission  effectiveness  and  appeared  to  be 
ideally  suited  for  this  type  of  strategic 
decision  (Keeney,  1973,  1976,  1992). 
The  approach  required  the  judgment  of 
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experts  regarding  future  values  and 
interpretation  of  organizational  mission 
and  goals.  The  type  of  decisions  the 
DFEB  typically  faces  appeared  to  be  a 
classic  budget  allocation  decision  across 
competing  objectives  since  the 
departments  were  so  inherently  different. 
Coincidentally,  the  Academy  was  in  the 
process  of  establishing  institution, 
education,  and  department  outcomes 
(largely  in  response  to  accreditation 
demands).  Thus,  the  time  was  ripe  to 
adapt  these  viable  targets  as  metrics  for 
performance.  The  DFEB  would  use 
these  metrics  to  develop  a  process  for 
funds  distribution  that  would  maximize 
the  effect  (or  value)  of  the  Academy’s 
educational  program  (or  mission). 

The  original  modeling  approach 
supported  by  the  Dean  was  an 
Equipment  Board  Hierarchical  Model 
(Francis,  1989).  The  fact  that  the  model 
survived  a  decade,  numerous  department 
heads,  and  rotating  DFEB  members 
lends  to  its  credibility.  Our  efforts  were 
to  validate  and  update  the  model  to 
create  a  closer  link  to  the  institutional 
values  and  mission  (thus  the  reliance 
upon  the  VET),  which  had  been 
reviewed  and  clarified  during  the 
intervening  decade.  Finally,  our  model 
had  to  incorporate  the  dynamics 
associated  with  a  government  fiscal 
environment.  Of  particular  interest  was 
its  flexibility  to  not  only  prioritize  initial 
budget  allocation,  but  also  to  address 
year-end  budget  residual  funds  when 
applicable.  In  either  case,  deriving  an 
objective  metric  of  how  much  positive 
impact  each  request  would  have  on  the 
mission  proved  to  be  a  very  powerful 
tool.  The  fact  that  this  approach  was  a 
potential  first  step  toward  GPRA 
compliance  and  organizational  goal 
congruence  provides  compelling 


rationale  for  other  organizations  to 
consider  its  virtues. 

HIERARCHY 

DEVELOPMENT 

Consensus  building  with  respect 
to  equipment  selection  criteria  was 
essential  to  the  success  of  the  hierarchy 
development.  Recognizing  this,  the 
DFEB  consisted  of  a  Chairman,  a 
Budget  Representative,  and  a  member 
from  each  of  the  four  academic 
divisions:  1)  Basic  Sciences,  2) 

Engineering,  3)  Social  Sciences,  and  4) 
Humanities.  Their  task  was  to  develop 
criteria  and  preference  weights  which 
supported  the  Academy’s  mission  and 
educational  outcomes,  as  well  as  to 
“level”  all  potential  parochial  division 
agendas. 

After  carefully  considering  the 
Academy’s  mission  and  educational 
outcomes,  the  DFEB  agreed  on  three 
main  criteria  for  equipment  purchases: 
1)  the  intended  equipment  usage,  be  it 
for  curriculum,  research,  or  support;  2) 
whether  the  equipment  type  served  a 
new  or  replacement  role;  and  3)  the 
individual  department’s  subjective 
desire  for  a  piece  of  equipment.  When 
first  developed,  the  board  included  a 
fourth  criterion:  whether  the  equipment 
had  been  planned  for  in  the  department’s 
budget.  However,  this  “budget” 
criterion  represented  only  a  procedural 
issue,  and  did  not  reflect  the  underlying 
values  of  the  institution.  Instead,  the 
board  recently  updated  this  criterion  to 
hazard,  which  reflected  the  more 
contemporary  issues  of  workplace  safety 
and  environment  regulations.  Figure  1 
depicts  the  updated  hierarchy. 

Under  the  usage  criterion,  the 
established  sub-criteria  of  curriculum, 
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research,  and  support  clarified  whether 
the  equipment  was  primarily  intended 
for  the  classroom,  faculty  research  and 
enrichment,  or  whether  it  fulfilled  some 
need  not  directly  related  to  academia. 
Below  each  sub-criterion,  the  attributes 
breadth  and  depth  provided  an 
opportunity  to  score  each  equipment 
request.  Depth  was  scored  by  the 
amount  of  equipment  usage,  given  in 
hours  or  lessons.  We  assigned  a  breadth 
score  based  on  how  many  courses  or 
departments  the  equipment  would  be 
used. 

The  type  criterion  was  broken 
into  replace  and  new  sub-criteria.  In  the 
case  of  equipment  to  be  replaced, 
categorical  descriptions  of  the  current 
equipment’s  condition  and  its 
technological  obsolescence  as 
measurable  attributes  were  used.  New 
equipment  was  measured  categorically 
by  its  enhancement  to  existing 
capability,  and  by  how  contemporary  the 
equipment’s  technology  was  with  respect 
to  its  intended  use. 

The  desire  criterion  allowed  each 
academic  department  to  prioritize  it’s 
own  equipment  requests.  This  was 
deemed  reasonable,  given  that  DFEB’s 
charter  was  to  prioritize  purchases  across 
diverse  disciplines,  not  within  a  given 
discipline.  Unlike  other  value  models, 
departmental  flexibility  was  “built-in.” 
For  example,  all  board  members  agreed 
that  autonomy  within  a  department  was 
prudent.  Each  department  received  a 
point  allocation  for  this  criterion  based 
on  their  number  of  requests.  A  single 
request  could  receive  100  points;  two 
requests  could  receive  only  a  total  of  180 
points.  However,  no  request  could  be 
assigned  more  than  100  points,  which 
thwarted  the  temptation  to  generate 


additional  requests  solely  to  shift  points 
to  a  genuine  need. 

The  DFEB  decided  that  fairness 
across  departments  could  be 
incorporated  into  the  desire  criterion.  In 
a  departure  from  traditional  VFT,  the 
DFEB  forced  each  department  to  rank 
order  their  requests  (prior  to  calculating 
the  value  function)  and  considered  the 
top  two  of  each  department  prior  to  all 
others.  This  “top-two”  rule  was  not 
arbitrary;  it  reflected  a  sense  of  equity  as 
well  as  realistic  upper  bound  on  the  total 
amount  of  equipment  the  budget  could 
support.  All  additional  requests  were 
ranked  below  the  combined  “top  two” 
lists  from  all  departments. 

Hazard  was  broken  into  sub¬ 
criteria  of  Environmental  and  Safety. 
These  were  scored  based  on  the 
categorical  level  of  redress  new 
equipment  provided.  Existing  violations 
of  law  or  regulation  received  the  highest 
priority;  impending  or  potential 
violations  received  lower  scores. 

Throughout  the  development 
process,  the  DFEB  carefully  considered 
the  Dean  of  Faculty  mission.  For 
example,  a  primary  educational  outcome 
of  the  Academy  is:  “Fundamental 
integrated  knowledge:  breadth  and 
depth.”  The  DFEB  used  this  and  other 
mission-oriented  statements  when 
designing  the  value  hierarchy  and  tree. 

The  levels  of  the  tree  depicted  in 
Figure  1  specify  the  primary  criteria,  the 
sub-criteria  (if  applicable),  and  the 
measurable  attributes.  Figure  1  also 
shows  the  weights  determined  by  the 
method  next  described. 

HIERARCHY  WEIGHTING 

After  reaching  consensus  on  the 
structure  of  the  tree,  the  DFEB  turned  to 
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establishing  the  relative  weights  for  the 
criteria,  sub-criteria,  as  well  as  assigning 
scores  for  each  measurable  attribute. 
Rather  than  utilize  the  pair-wise 
comparisons  suggested  by  AHP,  the 
DFEB  employed  a  modified  Delphi 
Process  and  directly  created  the 
hierarchy’s  attribute  weights.  This 
process  was  possible  since  board 
members  thoroughly  understood  the 
hierarchy,  the  weights  assigned  to 
attributes,  and  the  resulting  value 
function.  (Crawford  and  Williams,  1985, 
discuss  the  use  of  subjective  weighting 
matrices).  The  resulting  Delphi  Process 
accomplished  two  goals  for  the 
facilitators.  First,  the  process  provided 
the  vehicle  to  insure  each  board  member 
understood  the  hierarchical  process. 
Second,  the  process  allowed  members  to 
voice  their  opinions  without  reservation. 
Within  the  Delphi  Process,  each  member 
responded  by  evaluating  the  relative 
importance  of  each  criterion  and  sub¬ 
criterion.  For  example.  Figure  2  depicts 
the  worksheet  used  to  establish  the 
relative  importance  of  curriculum 
usage,  research  usage,  and  support 
usage. 

When  the  DFEB  initially 
convened,  three  iterations  were 
necessary  to  achieve  consensus.  Each 
iteration  consisted  of  soliciting 
individual  scores,  and  meeting  as  a 
group  to  present  those  individual  results 
(based  upon  Saaty’s  AHP  method, 
1982).  More  recently,  the  weight 
determination  proceeded  much  quicker, 
no  doubt  due  to  the  face-validity  of  the 
earlier  work,  as  well  as  the  VFT 
emphasis  upon  values  leading  to 
objectives.  The  DFEB  agree  on  all  the 
weights  after  the  first  iteration  of  the 
Delphi  Process. 


Once  the  DFEB  decided  on  the 
relative  weights,  normalizing  the  values 
so  that  each  criterion  level  of  the  tree 
summed  to  one  was  straightforward. 
Note  in  Figure  1  that  the  attribute  scores 
(in  contrast  to  the  criteria  weights)  do 
not  sum  to  one;  the  highest  score  for 
each  attribute  was  assigned  1;  lower 
scores  were  scaled  according  to  DFEB 
members’  assessment. 

SURVEY,  FINDINGS  AND 
RESULTS 

The  DFEB  developed  a  survey 
consisting  of  a  question  relating  to  each 
sub-criterion,  which  was  scored  by 
multiple  choice  response  (see  appendix). 
The  academic  departments  completed  a 
survey  for  each  of  their  equipment 
requests.  As  with  most  surveys,  there 
was  some  initial  confusion  as  to  how  to 
complete  the  questionnaire  (particularly 
with  respect  to  assigning  points  under 
the  desire  criterion).  Eventually,  each 
department  complied. 

Figure  3  lists  each  equipment 
request  and  its  corresponding  VFT  score. 
The  first  and  second  priority  requests  for 
each  department  are  sorted  separately  at 
the  top;  remaining  requests  are  sorted 
below  the  “top  two”  list. 

A  few  items  were  removed  from 
the  list  administrative  reasons  (e.g.,  the 
items  were  unavailable,  or  were  funded 
by  other  sources),  but  the  priority  was 
unchanged  for  the  remaining  ones. 
Ultimately,  items  with  a  score  greater 
than  28.0  were  funded  immediately, 
corresponding  to  just  over  $1  million. 
The  remaining  items  successfully 
competed  for  year-end  funds,  thanks  to 
the  mission  justification  offered  by  this 
process. 
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An  example  question  to  each  board  member: 

With  respect  to  Usage ,  please  assess  the  relative  importance 
of  Curriculum,  Research,  and  support. 

Curriculum  | . | . | . I . I 

Research  | . I . I . I  *  *  *  *  *  I 

Support  1 . 1 . | . I . I _ 


A  sample  response: 

With  respect  to  Usase.  please  assess  the  relative  importance 
of  Curriculum,  Research,  and  support. 

Curriculum  j . j . i  •  “ L  •  •  | . I 

Research  frrrrrt . 1  "’"•••  I . I 

Support  |  •  •  •  r*  | . 1 1 . 1 _ 


Figure  2  We  established  consensus  on  the  appropriate  weights  for  the  hierarchy 
tree  by  having  each  board  member  draw  horizontal  histobars  reflecting  relative 
importance.  In  this  case,  the  respondent  considered  curriculum  use  roughly  5/8 
as  important  as  research  use,  and  support  use  just  under  1/4  as  important  as 
research  use.  The  entire  board  was  presented  with  all  the  respondents’  answers; 
conflicts  were  resolved  by  the  Delphi  method. 


SENSITIVITY  ANALYSIS 

The  DFEB  was  concerned  of  the 
effect  that  their  specific  preference 
weights  would  have  upon  the  final  rank 
ordering  of  equipment  requests.  To 
address  this  concern,  we  altered  the 
preference  weights  of  the  four  primary 
factors  {usage,  desire,  tgge.  and  hazard ) 
first  by  10%,  then  20%,  and  evaluated 
the  resulting  rank  orderings.  The 
sensitivity  of  each  factor  was  performed 
independently.  For  instance,  when 
desire  was  altered  from  its  original  30% 
value  to  27%  (a  10%  change),  the 
preference  weights  of  the  remaining 
three  factors  increased  proportionately. 

The  results  were  somewhat 
unexpected.  Although  the  DFEB 
believed  hazard  to  be  an  important 
factor,  it  had  almost  no  impact  upon  the 
final  equipment  rankings.  As  seen  in 


Figure  4,  only  10%  of  the  equipment 
items  moved  within  the  overall  rankings 
despite  an  increased  preference  for 
hazard  from  15%  to  25%.  Based  upon 
this  information,  we  will  initiate  efforts 
to  improve  the  measurement  of  hazards. 
Similarly,  type  sensitivity  (Figure  5)  did 
not  significantly  affect  the  overall  item 
rankings.  Again,  future  efforts  will 
address  the  value-added  of  this 
dimension  (new  versus  replacement 
technology)  to  create  a  more 
discriminating  metric. 

On  the  other  hand,  usage  and 
desire  (Figures  6  and  7)  were  influential 
in  their  impact  on  the  final  rankings.  For 
example,  item  #33’s  position  was  very 
sensitive  to  desire  preference.  Its  rank 
ranged  from  30th  to  40th’  as  desire 
became  more  important  (d-20%  to 
d+20%).  This  piece  of  equipment  was 
important,  but  it  was  not  its  department’s 
first  choice.  Similarly,  items  #5  and  #10 
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Department 

Item 

Number 

Unit 

Total 

Cumulative 

Value  Tree 

and  Priority 

Description 

Requested 

Cost 

Cost 

Cost 

Score 

Number 

DFX 

STAND.  VI D 

10 

SXXX 

sxxx 

sxxx 

63.5 

DFX 

DIGITAL  OSCILLOSCOPES 

11 

SXXX 

sxxx 

sxxx 

47.2 

DFX 

CONTROLS  LAB  SERVO-MOTORS 

14 

SXXX 

sxxx 

sxxx 

46.7 

DFX 

CONVERTER,  VID 

7 

SXXX 

sxxx 

sxxx 

46.2 

DFX 

VESTIBULR  ILLUSION 

1 

SXXX 

sxxx 

sxxx 

45.3 

DFX 

DEMONSTRATOR 

BENCH  TOP  POWER  SUPPLIES 

6 

SXXX 

sxxx 

sxxx 

44.3 

DFX 

FINE  ELECTRON  BEAM  TUBES 

2 

SXXX 

sxxx 

sxxx 

44.3 

DFX 

FUNCTION  GENERATOR 

5 

SXXX 

sxxx 

sxxx 

44.2 

DFX 

SOIL  SAMPLING  RIG 

1 

SXXX 

sxxx 

sxxx 

40.2 

DFX 

XENETECH  ENGRAVER  TABLE 

1 

SXXX 

sxxx 

sxxx 

40.0 

DFX 

“Top  Two”  List  above,  all  other  requests  below- 
OVERHEAD  PROJECTOR  17  SXXX 

sxxx 

sxxx 

55.3 

DFX 

STAND.  PROJECT 

17 

SXXX 

sxxx 

sxxx 

50.7 

DFX 

WALK-IN  COLD  ROOM  UNIT 

1 

SXXX 

sxxx 

sxxx 

47.1 

DFX 

MOTION  CONTROLLER 

1 

sxxx 

sxxx 

sxxx 

46.0 

DFX 

CINEMA  CLASSICS  VIDEODISCS 

10 

SXXX 

sxxx 

sxxx 

42.8 

DFX 

FIBER  LAUNCH 

1 

sxxx 

sxxx 

sxxx 

42.5 

DFX 

DATA  ACQUISITION  EQUIPMENT 

2 

sxxx 

sxxx 

sxxx 

42.2 

DFX 

FATIGUE  TEST  MACHINE  UPGRADE 

1 

sxxx 

sxxx 

sxxx 

41.8 

DFX 

OPTICAL  ISOLATOR 

1 

sxxx 

sxxx 

sxxx 

41.6 

DFX 

FUNCTION  GENERATOR 

6 

sxxx 

sxxx 

SXXX 

41.1 

DFX 

VIBRATION  ISOLATION  LEG 

1 

sxxx 

sxxx 

SXXX 

40.6 

DFX 

OSCILLOSCOPE 

48 

sxxx 

sxxx 

sxxx 

40.5 

DFX 

COMPLETE  ROTATIONAL  SYSTEM 

34 

sxxx 

sxxx 

sxxx 

39.9 

DFX 

ELETROSTATICS  LABORATORY 

6 

sxxx 

sxxx 

SXXX 

38.5 

DFX 

SYSTEM 

VIDEO  IMAGING  SYSTEM 

1 

sxxx 

sxxx 

sxxx 

37.8 

DFX 

10KIP  LOAD  CELL 

1 

sxxx 

sxxx 

sxxx 

37.4 

DFX 

EQUIPMENT  TRAILER 

1 

sxxx 

sxxx 

sxxx 

37.1 

DFX 

ASPHALT  PAVING  MACHINE 

1 

sxxx 

sxxx 

sxxx 

36.2 

DFX 

EM-SCAN  ANALYSIS  SYSTEM 

1 

sxxx 

sxxx 

sxxx 

35.2 

DFX 

HECD LASER 

1 

sxxx 

sxxx 

sxxx 

31.4 

DFX 

SPACE  SIM  MAGNETIC  FIELD 

1 

sxxx 

sxxx 

sxxx 

31.2 

DFX 

DFX 

DFX 

DFX 

DFX 

DFX 

CONTROL 

CW  DOUBLED  YAG  LASER  SYSTEM 
FINE  ELECTRON  BEAM  TUBES 
WAREHOUSE  PALLET  JACKS 
HYDRAULICS  FLOW  BENCH 
UNCONFINED  COMPRESSION 

TESTERS 

SPACE  SIM  THERMAL  CONTROL 

EQUIP 

_  FUNDING  LINE - -  — 

1 

4 

2 

1 

2 

1 

SXXX 

sxxx 

sxxx 

SXXX 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

sxxx 

31.1 

30.8 

29.3 

29.1 

28.6 

28.2 

DFX 

DIRECT  SHEAR  TESTERS 

2 

sxxx 

sxxx 

sxxx 

27.1 

DFX 

NUCLEAR  DENSIOMETER-TROXLER 

1 

sxxx 

sxxx 

sxxx 

26.7 

DFX 

SPECTRUM  ANALYZER 

1 

sxxx 

sxxx 

sxxx 

26.6 

DFX 

PULSED  ENERGY  METER 

1 

sxxx 

sxxx 

sxxx 

26.2 

DFX 

LOCK-IN  AMPLIFIER 

1 

sxxx 

sxxx 

sxxx 

25.3 

DFX 

THERMO  AND  STATIS  MECHANICS 
DEMOS 

1 

sxxx 

sxxx 

sxxx 

24.7 

DFX 

MICROWAVE  OPTICS 

1 

sxxx 

sxxx 

sxxx 

24.3 

DFX 

EMISSIONS  TESTER 

1 

sxxx 

sxxx 

sxxx 

23.1 

DFX 

WEATHER  MONITORS 

1 

sxxx 

sxxx 

sxxx 

22.3 

DFX 

CARBON  MONOXIDE  METER 

1 

sxxx 

sxxx 

sxxx 

20.3 

DFX 

MAGNETIC  TORQUE 

5 

sxxx 

sxxx 

sxxx 

20.3 

DEMONSTRATOR 

Figure  3  The  requested  equipment  for  the  USAEA  Dean  of  Faculty  is  prioritized  by  department 
desire  and  total  value  tree  score.  The  first  and  second  most  desired  items  for  each  department  are 
given  highest  priority;  remaining  items  are  listed  by  score  only.  Items  above  the  funding  line 
received  immediate  funding,  while  the  remaining  items  successfully  competed  for  year-end  funds 


(department  names  and  costs  have  been  masked). 
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Rank  Order 
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TYPE  Sensitivity 


Range  of  TYPE  Weights 


Figure  5.  Changing  the  weight  of  the  type  criterion  did  not  significantly  affect  the  ranking  although 
it  had  more  effect  than  changing  hazard' s  weight.  Note  that  the  equipment  request  lines  cross  each 
other  more  frequently  than  figure  4.  Whether  an  item  was  new  technology  or  replacing  old 
technology  was  not  significant. 


Rank  Order 
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Range  of  USAGE  Weights 


Figure  6.  The  DFEB  considered  usage  a  prime  attribute  contributing  to  the  Academy  mission;  items 
not  used  within  the  classroom  fell  to  the  bottom  of  the  rankings.  This  effect  insures  future  requests 
are  very  classroom-friendly! 
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DESIRE  Sensitivity 


Range  of  Desire  Weights 


Figure  7.  This  figure  is  very  important  to  Department  Heads.  They  must  support  their  equipment 
item  requests,  and  the  Board  must  carefully  determine  how  much  power  they  want  the  Departments 
to  have.  The  ranking  of  several  items  plummeted  as  the  desire  weight  increased,  indicating  the  peril 
of  departmental  indifference! 
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scored  high  based  upon  the  hierarchy, 
but  were  not  their  department’s  priority 
items.  These  equipment  items  were 
victims  of  DFEB’s  decision  to  consider 
the  top  two  choices  of  each  department 
before  considering  other  items.  Item  #5 
was  its  department’s  3rd  choice,  even 
though  it  was  in  the  top  5  overall.  The 
sensitivity  analysis  provided  information 
to  members  of  the  DFEB  that  was  lost 
under  the  “top  2”  criteria.  The  DFEB 
remedied  this  inconsistency  by  funding 
these  items  first  as  “fallout”  funds  came 
available.  Since  the  process  is  intended 
to  avoid  tactics  which  eliminate 
potentially  valuable  equipment  items,  we 
recommend  that  the  board  refine  their 
desire  measurement  to  insure  it  captures 
the  essence  of  the  “top  2”  criteria. 


CONCLUSIONS  AND  FUTURE 
ISSUES 

Value-Focused  Thinking  (VFT) 
played  the  primary  role  in  the  Air  Force 
Academy’s  equitable  distribution  of 
budget  authority  to  widely  disparate 
academic  departments.  The  approach 
described  in  this  paper  provided  the 
DFEB  the  credible  information  required 
to  immediately  put  into  action  a 
performance-based  decision  and 
budgeting  process  that  considers  both 
internal  and  external  perspectives.  It 
also  dramatically  reduced  the 
emotionally  charged  debated  that 
historically  accompanied  the  DFEB’s 
process. 

Through  VFT,  the  DFEB  was 
able  to  allocate  resources  with  a  sound 
analytical  foundation  for  a  complex 
strategic  decision.  The  model’s  criteria 
were  based  on  the  institutional  values  of 


education,  mission,  adherence  to 
regulatory  imperatives,  and  a  sense  of 
fairness,  rather  than  “evenly  slicing  the 
budget  pie.”  With  institutional  values 
providing  the  aegis  for  legitimacy,  the 
model  overcame  traditional  budget 
contentions.  VFT  provided  a  rubric  for 
assessing  how  well  aligned  procurement 
decisions  were  to  the  Academy’s  values, 
mission,  and  objectives.  Thanks  to  the 
mission  impact  elucidated  by  our 
approach,  we  did  not  need  to  utilize  a 
post-processing  optimization.  In  the 
future,  as  others  adopt  this  method, 
benefit  to  cost  optimizations  will  be 
necessary. 

The  VFT  approach  incorporated 
the  dynamic  fiscal  environment  and 
provided  the  flexibility  to  not  only 
prioritize  initial  budget  allocation,  but  to 
also  address  year-end  budget  residual.  It 
allowed  this  organization  to  move 
toward  implementing  a  results-oriented 
budgeting  and  management  system, 
rather  than  focus  inappropriately  on  the 
cost  of  each  request.  Since  performance- 
based  budgeting  requirements  will 
undoubtedly  drive  the  decision-making 
landscape  for  the  foreseeable  future,  the 
decision  process  described  in  this  paper 
could  provide  insights  for  improving  the 
decision  processes  for  countless 
government  and  commercial 
organizations. 

Opinions,  conclusions,  and 

recommendations  expressed  or  implied 
within  are  solely  those  of  the  authors 
and  do  not  necessarily  represent  the 
views  of  the  United  States  Air  Force 
Academy,  the  United  States  Air  Force, 
the  Department  of  Defense,  or  any 
other  US  government  agency. 


9 


UNCLASSIFIED 


REFERENCES 

Bernhard,  R.,  “Some  Problems  in  Using 
Benefit/Cost  Ratios  with  the  Analytic  Hierarchy 
Process,”  The  Engineering  Economist  36  (1), 

Fall  1990,  Pages  56-65. 

Burk,  R.,  and  Parnell,  G.,  “Evaluating  Future 
Military  Space  Technologies,”  Interfaces  Voi  27, 
No  3,  1997. 

Comes,  W.,  and  Riley,  C.,  “Federal  Financial 
Statements:  The  Revolution  is  Here!,”  Journal  of 
Accountancy.  Volume  187,  Number  6,  June 
1999.  P.26. 

Crawford,  G.,  and  Williams,  C.,  “The  Analysis  of 
Subjective  Judgment  Matrices,”  RAND  Report 
R-2572-1-AF.  May  1985. 

Figura,  S.,  “Capital  Considerations.” 

Government  Executive.  February  1999,  p.26. 

Francis,  V.,  “DFEB  Hierarchical  Model,” 
Unpublished  Working  Paper,  USAF  Academy, 
CO,  1989. 

French,  S.,  Decision  Theory:  An  Introduction  to 
the  Mathematics  of  Rationality.  John  Wiley  & 
Sons,  NY,  1988. 

French,  S.,  Readings  in  Decision  Analysis, 
Chapman  &  Hall,  New  York,  NY,  1989. 

Gass,  S.,  “A  Process  for  Determining  Priorities 
and  Weights  for  Large-Scale  Linear  Goal 
Programs,”  OR:  The  Journal  of  the  Operational 
Research  Society.  Voi  37  (8),  1986. 

Government  Accounting  Office  (GAO), 
“Effectively  Implementing  the  Government 
Performance  and  Results  Act,”  GAO/GGD-9 6- 
118,  June  1998. 

Hague,  D.,  Newton,  H.,  and  Lowe,  J.,  “AFROTC 
Viability  Study,”  Presentation  at  66th  MORS 
Symposium,  Monterey,  CA,  Jun  1998. 

Hwang,  C.,  and  Yoon,  K.,  Multiple  Attribute 
Decision-Making:  Methods  and  Applications,  A 
State-of-the-Art  Survey.  Springer- Verlag,  Berlin, 
Germany,  1981. 


Keeney,  R.L.  “A  Decision  Analysis  with 
Multiple  Objectives:  The  Mexico  City  Airport,” 
Bell  Journal  of  Economics  and  Management 
Science  4.  1973. 

Keeney,  R.L.  and  Raiffa,  H.  Decisions  with 
Multiple  Objectives:  Preferences  and  Value 
Trade-offs,  John  Wiley  &  Sons,  Inc.,  New  York, 
NY,  1976. 

Keeney,  R.L.,  Value-Focused  Thinking:  A  Path 
to  Creative  Decision-Making.  Harvard 
University  Press,  Cambridge,  Mass,  1992. 

Kirkwood,  C.,  Strategic  Decision-Making:  Multi- 
Objective  Decision  Analysis  with  Spreadsheets, 
Duxbury  Press,  1997. 

Laurent,  A.,  “Results  or  Else.”  Government 
Executive.  June  97,  p.18 

Liberatore,  M.[l],  “An  Extension  of  the  Analytic 
Hierarchy  Process  for  Industrial  R&D  Project 
Selection  and  Resource  Allocation,  ”IEEE 
Transactions  on  Engineering  Management.  EM- 
34,  1987,  pp.  12-18. 

Liberatore,  M.[2],  “R&D  Project  Selection,” 
Telematics  and  Informatics  3(4).  1987,  pp.289- 
300. 

Melese,  F.,  Lowe,  J.,  and  Stroup,  M., 

“Integrating  Cost  and  Effectiveness:  an 
Economic  Perspective,”  Phalaax.  30GL 
September  1997,  pp.13-17. 

Melese,  F.,  and  Lowe,  J.,  “Downsizing 
Strategies,”  Navy  Comptroller,  Winter  1992. 

Mitchell,  K.,  and  Bingham,  G.,  “Maximizing  the 
Benefits  of  Canadian  Forces  Equipment 
Overhaul  Programs  Using  Multi-Objective 
Optimization,”  INFOR:  The  Canadian  Journal  of 
Operational  Research.  24f4L  1986,  pp.25 1-264. 

Office  of  Management  and  Budget  (OMB), 
“Preparing  and  Submission  of  Strategic  Plans 
and  Annual  Performance  Plans,”  Circular  No.  A- 
11,  Part  2,  July  1998,  p.21. 

Parnell, G.,  Conley,  H.,  Jackson,  J.,  Lehmkuhl, 

L.,  and  Andrew,  J.,  “FOUNDATIONS  2025:  A 
Value  Model  for  Evaluating  Future  Air  and 
Space  Forces,”  Management  Science  44(10). 

1998,  pp.1336-1380. 


11 


UNCLASSIFIED 


Saaty,  T.,  The  Analytic  Hierarchy  Process,  New 
York,  NY,  McGraw-Hill  Book  Company,  Inc, 
1980. 

Saaty,  T.,  Decision  Making  for  Leaders:  The 
Analytic  Hierarchy  Process  for  Decisions  in  a 
Complex  World,  Wadsworth,  1982. 

Saaty,  T.,  “How  to  Make  a  Decision:  The 
Analytic  Hierarchy  Process,”  European  Journal 
of  Operations  Research  48,  1990. 


Vazsonyi,  A.,  “Multicriteria  Decision  Making,” 
Decision  Line,  Dec  1994/Jan  1995. 


UNCLASSIFIED 


APPENDIX 


DF  EQUIPMENT  BOARD 
EQUIPMENT  REQUEST  QUESTIONAIRE 

1.  How  many  TOTAL  LESSONS  per  SEMESTER  will  the  requested  equipment  be  used?  Count 
ALL  courses  however  count  only  once  per  course.  For  example;  EE350,  EEj51,  and  EE352  each  use 
an  oscilloscope  for  40, 35,  and  20  lessons  each,  totaling  95. 

a.  More  than  40  Lessons. 

b.  More  than  30  Lessons. 

c.  More  than  20  Lessons. 

d.  More  than  10  Lessons. 

e.  Less  than  10  Lessons. 

f.  Not  used  in  class. 

2.  In  which  TYPES  of  courses  will  the  requested  equipment  be  used? 

a.  CORE  and  MAJOR’S  Courses. 

b.  CORE  Courses 

c.  MAJOR’S  Courses  (include  Service  Courses  for  other  Departments) 

d.  Not  used  in  class. 

3.  How  much  will  the  equipment  you  are  requesting  be  used  to  support  RESEARCH? 

a.  More  than  30  hours  per  week. 

b.  Between  21  and  30  hours  per  week. 

c.  Between  1 1  and  20  hours  per  week. 

d.  Between  5  and  10  hours  per  week. 

e.  Less  than  5  hours  per  week. 

f.  Not  used  in  RESEARCH. 

4.  This  equipment  will  support  RESEARCH  in  how  many  departments/agencies? 

a.  Four  or  more. 

b.  Three 

c.  Two 

d.  One 

e.  Not  used  in  RESEARCH. 

5.  NOT  including  use  for  COURSES  and  RESEARCH,  how  much  will  this  equipment  be  used? 

a.  More  than  30  hours  per  week. 

b.  Between  21  and  30  hours  per  week. 

c.  Between  1 1  and  20  hours  per  week. 

d.  Between  5  and  10  hours  per  week. 

e.  Not  used  for  other  than  CURRICULUM  or  RESEARCH. 

6.  NOT  including  use  for  COURSES  and  RESEARCH,  how  many  agencies  will  use  this  equipment? 

a.  Common  use  item  for  most/all  of  DF,  34TRW ,  SUPT  agencies 

b.  More  than  three  departments 

c.  Three  departments/agencies 

d.  Two  departments/agencies 

e.  One  department/agency 

f.  Not  used  other  than  in  the  CURRICULUM  and/or  research 

QUESTIONS.  7  AND  8  APPLY  TO  REPLACEMENT, EQUIPMENT. ITEMS. 

QUESTIONS  9  AND  IQAPPLYTO  NEW  EQUIPMENT  ITEMS. 

FOR  RE  PL  A  CEMENT  ITEMS:  PLEASE  BE  SURE  TO  ANSWER  “c”  and  “d”  TO 
QUESTIONS  9  AND  /ft 


FOR  NEWITEMS:  PLEASE  BE  SURE  TO  ANSWER  “f’  and  “c”  TO  QUESTIONS.  7  AND  S. 


UNCLASSIFIED 


7.  Which  best  describes  the  CONDITION  of  the  equipment  being  REPLACED: 

a.  Inoperable,  impossible  or  too  expensive  to  repair 

b.  Operable,  but  performance  is  significantly  degraded 

c.  Operable,  but  frequent  repair  is  required 

d.  Operable,  but  has  exceeded  expected  life  cycle 

e.  Operable,  but  the  equipment  to  be  purchased  is  much  better 

f.  Not  a  replacement  equipment  item 

8.  How  would  you  best  describe  the  technology  of  the  equipment  being  replaced? 

a.  Item  being  replaced  is  of  outmoded  technology;  replacement  item  wills  significantly  upgrade 
performance. 

b.  Item  being  replaced  is  of  current  technology;  replacement  item  will  be  of  approximately  the 
same  technology  and  have  approximately  the  same  capabilities. 

c.  Not  a  replacement  equipment  item. 

9.  How  will  the  requested  NEW  EQUIPMENT  ITEM  improve  CAPABILITY? 

a.  Provides  new  capabilities:  allows  the  using  agencies  to  do  things  they  are  currently  unable  to 
do. 

b.  Provides  expanded  capabilities:  allows  the  using  agencies  to  do  more  of  what  they’re  doing 
now,  or  do  it  more  efficiently. 

c.  Not  a  NEW  EQUIPMENT  ITEM 

10.  How  would  you  best  describe  the  ‘technology’  of  this  NEW  EQUIPMENT  ITEM? 

a.  Current  technology:  proven  performance,  pretty  much  the  “standard” 

b.  New,  proven  technology:  new  on  the  market,  and  proven  technology,  state-of-the-art. 

c.  Not  a  NEW  EQUIPMENT  ITEM 


11.  Purpose:  In  question  11,  you  are  asked  to  allocate  “IMPORTANCE  POINTS”  to  each  of  your 
equipment  requests.  The  purpose  is  for  YOU  to  assess  how  “IMPORTANT”  this  request  is  relative  to  the 
others  you  have  submitted.  It  also  serves  to  help  equalize  the  budget  allocation  among 
departments/agencies. 

Instructions: 

1.  Determine  how  many  requests  your  department/agency  has  submitted. 

2.  From  the  table  below,  determine  how  many  POINTS/REQUESTS  you  are  allowed. 

3.  Determine  the  TOTAL  POINTS  you  have  to  allocated  by  multiplying  (POINTS/REQUEST)  X 
(NUMBER  OF  REQUESTS). 

4.  Allocate  the  TOTAL  POINTS  among  your  requests  to  indicate  their  RELATIVE  IMPORTANCE  (or 
DESIRE  ABILITY).  Do  not  allocate  more  than  100  POINTS  to  any  single  request. 

POINTS/REQUEST  TABLE: 

NUMBER  OF  REQUEST  POINTS/REQUEST 


1 

100 

2  to  4 

90 

5  to  7 

80 

8  to  10 

70 

llto  15 

60 

Please  fill  in  the  blank  with  the  point  allocation  for  this  equipment  item. 
POINTS= _ 
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12.  The  Requested  Equipment  remedies  a: 

a.  Direct  Violation  of  NON-WAIVERABLE  policy  (attach  documentation).” 

b.  Direct  Violation,  but  presently  WAIVERABLE  policy  (attach  documentation).” 

c.  Violation  of  an  IMPENDING  policy,  or  regulation  (attach  documentatation).” 

d.  Not  an  environmental  concern  at  this  time.” 

13.  The  Requested  Equipment  remedies: 

a.  A  Documental  Safety  write-up  (attach  BASE  SAFETY  OFFICE  documentation).” 

b.  A  Potential  Safety  hazard  (attach  DEPARTMENT  or  BASE  SAFETY  documentation).” 

c.  No  apparent  Safety  Hazard.” 


THANK  YOU  FOR  YOUR  ANSWERS! 
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Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining  the 
data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for  reducing 
this  burden  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188).  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202- 
4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a  currently 
valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 
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